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1 .  INTRODUCTION 

A  new  awareness  and  interest  in  statistical  problems 
encountered  in  the  design  and  analysis  of  weather  modification 
experiments  have  developed  recently,  perhaps  stimulated  by 
Brillinger,  Jones  and  Tukey,  hereafter  BJT,  (1978).  Sessions 
on  statistics  in  weather  modification  have  been  included  in 
recent  meetings  of  statistical  societies,  a  Tallahassee  Work¬ 
shop  on  Weather  Modification  sponsored  jointly  by  the  Office 
of  Naval  Research  and  the  Florida  State  University  was  held  in 
November,  1978,  and  a  two-number,  special  issue  of  Coimuni- 
cations  in  Statistics ,  Theory  and  Methods  was  devoted  to  the 
statistical  analysis  of  weather  modification  experiments  in 
1979.  Braham  (1979)  led  discussion  of  field  experimentation 
in  weather  modification  with  comments  added  by  a  number  of 


workers  in  the  field. 


BJT  expressed  grave  concerns  about  the  validity  of  para¬ 
metric  analyses  of  weather  modification  experiments.  Gabriel 
(1979,  p.  984,  985)  provides  two  quotations  from  their  report 
and  we  quote  from  his  paper  to  show  that  he  has  even  more 
concerns: 

"...Statisticians  have  indeed  been  brave  to 
risk  using  simple  parametric  models  and  methods 
of  inference  in  the  analysis  of  weather  modifi¬ 
cation  experiments.  These  parametric  methods 
are  quite  unlikely  to  have  been  valid  when  the 
underlying  assumptions  were  so  patently  untrue." 

"...However,  other  workers  have  continued 
to  use  parametric  methods,  evidently  preferring 
to  ignore  their  dubious  validity.  One  may  hope 
that  the  clear  statement  by  the  Task  Force 
(BJT,  1978)  might  put  an  end  to  this." 

At  the  noted  Tallahassee  Workshop,  exploratory  parametric 
analyses  of  data  from  Phase  I  of  the  Santa  Barbara  Con¬ 
vective  Seeding  Test  Program  were  presented,  Bradley, 
Srivastava  and  Lanzdorf ,  BSL,  (1979) .  (This  test  program 
in  two  phases  was  designated  Santa  Barbara  II  by  BSL.) 

Several  discussants  criticized  the  use  of  parametric  methods 
with  varying  degrees  of  fervor. 

The  purpose  of  this  article  is  to  respond  to 
criticisms  of  parametric  analyses,  particularly  if  used  in 
approximation  to  randomization  analyses.  Our  objective  is 
to  put  the  issues  in  perspective  and  it  is  not  in  any  way  to 
obviate  the  responsibilities  of  those  who  devise  stochastic 
models  and  statistical  analyses  to  examine  the  validity  of 
assumptions  inherent  in  their  methods. 

We  conclude  the  article  with  some  comnents  on  multi¬ 
plicity  of  analyses,  always  pertinent  in  reanalyses  of  exper¬ 
iments,  as  discussed  by  BJT  (1978,  Section  33),  Gabriel  (1979, 
Section  4),  and  Kruskal  (1979,  Section  3). 


2.  THE  SANTA  BARBARA  EXPERIMENT 

BSL  (1979)  give  a  summary,  with  references,  of  Phase  I  of 
the  Santa  Barbara  experiment.  Winter  storms  in  the  region  have 
convective  cells  grouped  into  identifiable  convective  bands, 
several  of  which  may  occur  in  a  storm.  The  convective  band 
was  used  as  the  experimental  unit.  Criteria  for  the  "seeda- 
bility"  of  a  convective  band  proved  to  be  only  partially 
operationally  usable.  A  predetermined  randomized  decision  to 
seed  or  not  seed  an  experimental  unit  was  made.  A  target 
area  and  a  control  area,  the  latter  West  of  the  target  area 
and  seeding  site,  were  defined.  Precipitation  responses 
attributable  to  each  experimental  unit  in  both  areas  were 
measured  through  use  of  networks  of  raingages,  not  all  of  which 
were  operative  for  every  experimental  unit.  Meteorological 
covariates,  some  of  them  measured  by  radiosonde,  were  observed 
for  each  convective  band;  unfortunately,  radiosonde  observa¬ 
tions  were  taken  at  Santa  Barbara  Airport  in  the  target  area. 

The  original  statistical  analyses  were  based  on  applica¬ 
tion  of  the  Wilcoxon -Mann -Whitney,  two-sample,  rank  test 
separately  for  the  data  from  each  raingage.  Both  precipitation 
measurements  and  ratios  of  precipitations  at  the  raingage 
divided  by  average  corresponding  control  area  precipitations 
were  used  with  similar  results.  These  analyses  were  reported 
by  Elliott  and  Thompson  (1972)  and  Brown,  Thompson  and  Elliott 
(1975).  Covariates  were  used  only  in  a  minor  way  to  consider 
classified  subsets,  or  separations  into  subsets,  of  the  exper¬ 
imental  data.  Concerned  with  both  the  validity  of  the  assump¬ 
tion  of  independence  of  successive  observations  for  the  non- 
parametric  tests  and  dependencies  among  the  tests,  Elliott 
and  Brown  (1971)  conducted  a  limited  Monte  Carlo  study  using 
their  data  with  randomized  relabelling  of  the  seeding  indica¬ 
tor.  They  report: 
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"At  the  0.05  significance  level  for  all 
bands,  29  stations  (raingages)  in  the  original 
test  sample  were  found  to  show  a  positive 
difference  between  seeded  and  not -seeded  cases; 
and  three  Mante  Carlo  runs  (out  of  50)  were 
found  to  have  as  high  or  higher  counts  of 
stations  with  a  positive  difference  at  this 
significance  level." 

Research  on  statistics  in  weather  modification  at  the 
Florida  State  University  had  two  objectives.  Independent 
reanalysis  of  the  Santa  Barbara  data  was  desired  because  of 
the  economic  importance  to  both  agriculture  and  energy  gener¬ 
ation  of  enhanced  precipitation  from  West  Coast  winter  storms. 
Improved  statistical  methodology  or  new  use  of  existing  pro¬ 
cedures  could  improve  similar  future  experimentation. 

Summarization  of  data  from  a  raingage  network  was 
considered  first.  BSL  (1977)  used  response  surfaces,  but 
found  that  their  "integrated  rainfall"  was  so  highly  corre¬ 
lated  with  the  simple  average  of  rainfall  responses  over 
raingages  in  a  target  or  control  area  that  the  average  was 
the  appropriate  summary  measure.  Scott  (1979)  used  principal 
components  for  similar  summarization  with  limited  success  and 
difficulties  in  application.  His  first  component  was  highly 
correlated  with  the  average  and,  although  two  additional 
components  could  be  identified  and  interpreted,  they  con¬ 
tributed  little  in  explanation  of  variability  among  rain¬ 
gages  in  a  target  or  control  area.  BSL  (1979)  examined  the 
usefulness  of  meteorological  covariates,  such  as  air -mass 
characteristics,  control  area  precipitations  and  band  passage 
times,  or  subsets  of  them,  along  with  other  possible  sources 
of  variation,  for  example,  storm  effects.  They  used  para¬ 
metric  analyses  after  data  transformation;  an  examination  of 
plots  of  standard  deviations  of  raingage  readings  against 
corresponding  means  for  each  experimental  unit  or  band  showed 
a  linear  relationship  and  suggested  the  transformation, 
z  =  log(y  +  A),  where  y  is  the  precipitation  at  a  raingage 
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for  a  band  and  A  is  a  constant  for  a  defined  target  area. 

The  use  of  covariates  essentially  eliminated  any  apparent 
effect  due  to  seeding  and  covariate  measures  were  suspected 
to  have  been  affected  by  seeding.  Simple  analyses  of  the 
effects  of  seeding  for  the  defined  target  area  and  subsets 
of  it  were  consistent  with  the  suggested  significance 
reported  in  the  quotation  above  from  Elliott  and  Brown  (1971) . 
Serfling  (1977)  used  the  covariate  data  to  group  responses 
for  bands  and  applied  nonparametric  methods  to  study  the 
effect  of  seeding  for  bands  within  groups  with  inconclusive 
results. 

BSL  (1979)  provided  analyses  of  variance  and  covariance 
for  various  target  areas  based  on  weighted  analyses  of  data 
transformed  as  indicated  above.  Target  areas  were  defined  in 
their  Table  I.  The  model  used,  (4.1)  in  the  reference,  had 


z  =  3, 


i=l 


B.V.  + 
1  x 


6Z  + 


where  z  is  the  mean  of  the  transformed  precipitation  responses 
for  raingages  in  a  designated  target  area,  Bn  is  a  general 
constant,  is  the  observation  on  the  l  covariate,  with  3^ 
being  the  corresponding  regression  coefficient,  i  =  1,  ...»  p, 
the  number  of  covariates  used  (all  may  be  omitted) ,  Z  =  1  or  0 
as  the  experimental  unit  (band)  was  or  was  not  seeded,  6  is  the 
parameter  measuring  the  additive  effect  of  seeding,  and  e  is  a 
random  error.  Weighted  least  squares  analyses  minimized 


N 

l,  V1.  •  60 

a=l 


i-1 


B.V. 

1  la 


-  6Z  ) 
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where  wq  was  the  number  of  raingage  observations  contributing 

to  z  in  band  a  and  N  was  the  total  number  of  bands,  some 
a 

seeded,  some  not.  Tables  I  and  II  below  are  taken  from 
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Tables  V  and  A-II  of  the  reference.  They  are  analyses  for 
Target  Area  (i),  the  main  planned  target  area;  other  target 
areas  involved  subsets  of  data  used  for  Target  Area  (i)  and 
gave  very  similar  results.  No  covariates  were  used  in  Table 
I  and  the  covariates  for  Table  II  were  those  of  Model  (6) 
in  (4.5)  of  the  reference.  In  the  next  section,  we  report 

TABLE  I 

Analysis  of  Variance  without  Covariates 
for  Target  Area  (i).  Transformed  Data 


Source  of 

d.f. 

Mean 

F- 

Variation 

Squares 

Ratio 

Seeding 

1 

110.29 

2.77 

Residual 

104 

39.84 

— 

TABLE  II 

Analysis  of  Covariance,  Reference  Model  (6) , 
for  Target  Area  (i).  Transformed  Data 


Source  of 
Variation 

d.f. 

Mean 

Squares 

F- 

Ratio 

Seeding  (adj.) 

1 

0.00 

0.00 

Covariates 

7 

374.40 

22.24 

Residual 

97 

16.83 

-- 

on  randomization  analyses  to  check  the  validities  of  the 
analyses  of  Tables  I  and  II.  Note  that,  in  the  reference, 
the  authors  avoided  the  probably  untenable  parametric 
assumptions  of  independence  and  normality  of  the  random 
errors  e  in  the  model.  In  addition,  the  weights  wq  were 


used  as  approximately  proportional  to  the  variances  of  the  eQ, 
although  no  one  would  assume  that  precipitations  at  nearby 
raingages  are  likely  to  be  independent.  The  question  is 
whether  or  not  the  parametric  analyses  are  adequate  approxi¬ 
mations  to  randomization  analyses  for  practical  purposes. 

The  question  is  not  whether  or  not  parametric  assumptions  are 
correct . 

3.  THE  RANDOMIZATION  TESTS 

The  randomization  distributions  of  F  were  sampled  by 
Monte  Carlo  methods  separately  for  the  data  of  Tables  I 
and  II.  The  number  of  Monte  Carlo  runs  for  each  table  was 
500.  The  sampling  mimicked  the  original  experimental  rando¬ 
mization;  a  value  from  a  symmetric  Bernoulli  distribution  was 
generated  for  each  run  and  thus  a  value  of  Z  in  the  model  was 
determined.  Accordingly,  the  numbers  of  declared  seeded  and 
not  seeded  experimental  units  were  observations  from  a 
symmetric  binomial  distribution  with  sample  size  106,  the 
number  of  bands  in  the  experiment.  For  each  run,  analysis 
like  that  of  Table  I  or  Table  II  was  done  and  the  resultant 
value  of  F  was  recorded  together  with  the  sign  of  the  esti¬ 
mated  seeding  effect. 

Sample  frequency  distributions  of  t,  the  signed  square 
root  of  F,  are  given  in  Table  III  for  the  data  of  Tables  I 
and  II.  It  is  clear  that  these  distributions  are  approximately 
symmetric  about  the  origin.  They  approximate  Student  distri¬ 
butions  with  104  and  97  degrees  of  freedom  respectively, 
although  these  Student  distributions  are  not  distinguishable 
for  practical  purposes.  The  mean,  variance,  and  coefficients 
of  skewness  and  kurtosis  of  t  for  the  data  from  Table  I  are 
in  order  0.029,  1.107,-0.049  and  -0.046  and  the  corresponding 
values  for  the  Student  distribution  with  104  degrees  of 
freedom  are  0,  1.020,  0,  and  0.060.  Similarly,  in  the  same 


7 


order  for  Table  II,  the  sample  values  are  -0.040,  1.033,  0.142, 
and  0.312  whereas,  for  the  Student  distribution  with  97  degrees 
of  freedom,  the  values  are  0,  1.021,  0,  and  0.065. 

In  assessment  of  significance  of  the  data  of  Table  I, 
a  one-sided  test  for  seeding  effect  seems  appropriate.  The 
500  Monte  Carlo  runs  resulted  in  60  values  of  F  not  exceeded 
by  the  observed  value  of  2.77,  37  of  them  with  apparent  posi¬ 
tive  effect  of  seeding  and  23  of  them  with  apparent  negative 
effect.  Since  we  know  that  the  complete  randomization  distri¬ 
bution  of  t  must  be  symmetric  about  the  origin,  the  proper, 
estimated,  one-sided  significance  level  is  Js(60/500)  =  0.060; 
the  corresponding  0.95  confidence  interval  is  (0.039,  0.081). 

TABLE  III 

Frequency  Distributions  of  t  for  500  Samplings 
of  the  Randomization  Distributions  of  the 
Data  of  Tables  I  and  II 


TABLE  I 

TABLE  II 

Class 

Interval 

Frequency  Relative 
Frequency 

Frequency  Relative 
Frequency 

Less  than-3.0 

2 

0.4 

2 

0.4 

-3,0  to  -2.5 

2 

0.4 

1 

0.2 

-2.5  to  -2,0 

13 

2.6 

9 

1.8 

-2,0  to  -1,5 

14 

2.8 

27 

5.4 

-1.5  to  -1.0 

49 

9.8 

47 

9.4 

-1.0  to  -0.5 

78 

15.6 

68 

13.6 

-0.5  to  0 

91 

18.2 

109 

21.8 

0  to  0.5 

89 

17.8 

104 

20.8 

0.5  to  1.0 

69 

13.8 

60 

12.0 

1,0  to  1.5 

48 

9.6 

38 

7.6 

1.5  to  2.0 

31 

6.2 

22 

4.4 

2.0  to  2.5 

11 

2.2 

6 

1.2 

2.5  to  3.0 

3 

0.6 

6 

1.2 

Gr.  than  3.0 

0 

0.0 

1 

0.2 

Totals 

500 

100.0 

500 

100.0 
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The  parametric,  one-sided  significance  level  from  tables  of 
the  Student  distribution  with  104  degrees  of  freedom  is  0.050. 
It  appears  that  the  parametric  method  has  given  a  satisfactory 
approximation  to  the  significance  level  of  the  appropriate 
randomization  test. 

Hie  mean  square  for  seeding  after  adjustment  for  the 
covariates  in  Table  II  is  remarkably  small.  The  covariates 
have  accounted  for  a  major  portion  of  the  experimental  varia¬ 
tion  and  the  residual  mean  square  has  been  reasonably  reduced. 
This  was  discussed  in  the  reference  where  the  authors  con¬ 
jectured  that  the  covariates  had  been  affected  by  treatment 
(seeding) ;  recall  that  covariates  were  measured  in  the  target 
area.  In  the  500  Monte  Carlo  runs  with  the  data  of  Table  II, 

12  values  of  F  (for  seeding)  were  recorded  that  did  not  exceed 
the  value  of  Table  II,  zero  when  rounded  as  recorded.  This 
study  seems  simply  to  support  the  earlier  conjecture. 

Analyses  like  those  of  Tables  I  and  II  were  very  similar 
for  other  target  areas  of  the  reference.  Monte  Carlo  studies 
can  be  expected  to  yield  comparable  results. 

4.  REMARKS 

Has  fortune  favored  the  brave?  (See  the  first  quotation 
from  Gabriel.)  We  think  not.  The  parametric  Student  test 
has  been  shown  to  be  relatively  robust.  The  authors  judged 
that  the  parametric  analyses  would  be  adequate  for  practical 
purposes  from  considerable  personal  experience.  Kempthome 
(1975,  p.  323),  in  discussion  of  the  role  of  normal -law 
parametric  theory,  states: 

"This  role  is  to  provide  useful  approxi¬ 
mations  to  the  randomization  test  procedure..." 

Miller,  Shaw  and  Veitch  (1979)  would  prefer  randomization  tests, 

but  in  consideration  of  a  cloud-seeding  experiment  in  Tasmania, 

both  consider  data  transformation  and  agreements  of  parametric 
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and  randomization  tests  as  reported  in  their  Table  II.  They 
state: 

'Table  II  shows  that,  in  this  experiment, 
the  significance  levels  obtained  from  the  per¬ 
mutation  test  agree  well  with  those  from  the 
F-test  without  pooling  which  suggests  that  the 
assumptions  of  the  classical  test  are 
adequately  satisfied." 

(We  would  have  preferred  a  conclusion  in  terms  of  adequacy 
of  approximation  of  the  F-test  to  the  permutation  test.) 
Significances  obtained  from  the  data  of  Table  I  are  in 
reasonable  agreement  with  the  small  permutation  test  sampling 
of  Elliott  and  Brown  (1971) , 

How  should  statisticians  proceed  with  the  analysis  of 
weather  modification  experiments?  The  cautious  statement 
by  BJT  has  our  partial  endorsement.  It  was  quoted  by  Gabriel 
(1979)  and  is  repeated  here: 

"...if  either  theory  or  substantial  experience 
shows  that  the  tail  areas  (degrees  of  unlikeliness) 
offered  by  an  approximation  to  a  parametric 
analysis  (we  usually  have  to  settle  for  approxi¬ 
mations  in  such  analyses)  are  repeatedly  and  con¬ 
stantly  close  to  the  results  of  re-randomization, 
it  may  be  wise  to  use  the  approximate,  parametric, 
tail  areas  even  more  widely,  reserving  re- 
randomization  calculations  for  only  the  most 
crucial  comparisons. " 

We  would  be  braver  than  BJT.  Exploratory  analyses  of  the 
limited  and  expensive  available  data  from  weather  modifi¬ 
cation  experiments  can  be  justified  as  a  search  for  new 
insights  subject  to  possible  confirmation  in  future  exper¬ 
iments.  This  is  the  way  that  science  works.  Creation  of  a 
need  to  use  only  re-randomization  analyses  is  likely  to 
stifle  such  investigations;  while  modem  computer  power  may 
make  such  analyses  possible,  they  are  cumbersome  and  expensive. 
We  believe  that  parametric  analyses  should  be  satisfactory 
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for  data  exploration  and  exploratory  use  of  new  means  of  data 
summarization  and  we  would  reserve  re -randomization  calcula¬ 
tions  for  confirmation  of  apparently  interesting  effects 
suggested  by  the  parametric  analyses.  In  this  way,  some  of 
the  experience  required  by  BJT  may  accumulate.  We  do  urge 
care  in  the  parametric  analyses  and  good  data  handling, 
including  use  of  transformations  when  indicated.  We  would 
endorse  the  caution  of  BJT  in  preplanned  data  analyses  of 
confirmatory  experiments. 

We  are  in  general  agreement  with  the  warnings  expressed 
in  the  references  of  Section  1  on  the  matter  of  multiplicity 
of  analyses.  BJT  (1978,  Section  33)  take  a  moderate  approach: 

"We  should  strive  to  do  as  much  inquiry  into 
alternate  modes  of  analysis  as  we  can  (a)  on  older 
data,  or  (b)  during  the  exploratory  phase  of  the 
study  that  most  concerns  us.  A  few  alternatives, 
particularly  if  highly  correlated  in  their  results, 
may  survive  into  confirmatory  phases,  but  we  should 
do  whatever  we  can  to  keep  the  nuirber  that  survive 
as  close  to  one  as  we  can." 

Gabriel  (1979,  Section  4)  elaborates  and  Kruskal  (1979, 

Section  3)  discusses  the  situation  with  examples,  suggesting 
that 

"Examples  of  the  high  frequency  of  low 
frequency  events  abound." 

Gabriel  considers  also  the  complementary  problem,  the  artificial 
inhibition  of  significance: 

"But  we  must  equally  warn  of  the  pitfall 
of  multiplying  the  number  of  re -analyses  of  a 
successful  experiment  (i.e.,  one  which  has  es¬ 
tablished  an  effect)  until  some  re-analysis 
reveals  another  correlate  of  the  "effect"  which 
may  then  be  used  as  an  alternative  explanation 
or  a  rejection  of  the  finding  that  the  treatment 
was  effective." 

He  used  analyses  later  reported  by  BSL  (1979)  to  illustrate: 
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"An  apparently  significant  precipitation 
difference  was  found  between  seeded  and  unseeded 
bands.  Subsequently,  in  the  course  of  a  large 
number  of  further  analyses,  the  seeded/unseeded 
difference  was  found  to  be  largely  confounded 
with  differences  between  storms,  with  no  signi¬ 
ficant  difference  remaining  between  seeded  and 
unseeded  bands  within  storms." 

Two  major  points  can  be  made  on  the  issue  of  multiplicity 
of  analyses.  The  first  is  that  the  degree  of  dependency  or 
correlation  is  crucial.  On  one  extreme,  repeated  application 
of  the  same  test  to  the  same  data  would  cause  no  concern, 
although  it  would  be  silly.  On  the  other  extreme,  searches 
for  some  subset  of  the  experimental  data  for  some  result  of 
significance  without  regard  to  experimental  objectives  or 
design  would  be  ludicrous  and  some  low-frequency  event  would 
surely  be  detected.  The  second  point  is  that  experimental 
design  itself  may  dictate  a  sequence  of  analyses  as  sources 
of  variation  and  improved  experimental  precision,  explicitly 
or  implicitly  designed  into  the  experiment,  are  considered 
for  their  effectiveness. 

Reanalysis  of  the  Santa  Barbara  experiment  was  initiated 
at  the  Florida  State  University  because  there  was  concern  with 
the  methods  originally  used  and  in  the  hope  that  new  insights 
for  improved  future  similar  experimentation  might  result. 

The  sequence  of  analyses  performed  represents,  in  our  view,  a 
considered,  logical  and  independent  reanalysis  of  the  Phase  I 
Santa  Barbara  data.  Further,  the  Phase  II  datawere  available 
and  reserved  for  possible  confirmatory  analyses,  analyses  that 
are  in  progress,  although  experimental  design  changes  lead 
to  new  difficulties.  The  sequence  of  analyses  began  with 
investigation  of  improved  methods  of  raingage  data  summarization. 
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Possible  covariates  were  considered  and  found  suspect,  seemingly 
having  been  affected  by  treatment.  A  transformation  was 
developed  for  improved  parametric  analyses.  Several  target 
areas  were  considered  and  yielded  highly  correlated  analyses; 
this  was  necessary  to  consider  in  a  limited  way  possible 
areas  of  effect  and  particularly  to  investigate  possible 
effects  far  downwind  as  reported  in  the  literature.  It  was 
clear  that  storm  effects  were  partially  confounded  with 
treatment.  We  believe  that  this  further  analysis  reported 
by  Gabriel  led  to  very  low  power,' within- storm  comparisons 
of  seeded  and  unseeded  experimental  units.  The  experimental 
design,  the  procedure  for  the  randomized  assignment  of 
treatment  to  experimental  units,  and  the  possible  effects  of 
treatment  on  otherwise  likely  covariates,  suggest  that  analyses 
like  those  of  Table  I  are  the  appropriate  analyses.  The  results 
of  sampling  the  randomization  distribution  of  t  reported  in 
this  article  substantiate  the  approximate  results  of  the 
parametric  analysis  of  Table  I. 

We  are  gratified  that  there  has  been  a  new  interest  by 
statisticians  in  the  design  and  analysis  of  weather  modification, 
experiments.  Needs  for  new  considerations  in  the  foundations  of 
statistical  inference  have  been  revealed  or  emphasized.  There 
are  similar  needs  for  new  and  improved  insights  into  cloud 
physics  and  experimental  procedures  from  meteorologists. 
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